Head-mounted device with active optical foveation

ABSTRACT

A pass-through camera in a head-mounted device may capture image data for displaying on a display in the head-mounted device. However, only low-resolution image data may be needed to display low-resolution images in the periphery of the user&#39;s field of view on the display. Therefore, the pass-through camera may only capture high-resolution images that correspond to the portion of the user&#39;s field-of-view that is being directly viewed and may capture lower resolution image data that corresponds to the real-world objects in the user&#39;s peripheral vision. To enable the camera module to selectively capture high-resolution images, the pass-through camera may include an image sensor with two or more pixel densities, a distortion lens, and/or one or more planar or curved mirrors. Any of the components in the camera module may be adjusted to change which portion of a scene is captured with high-resolution image data.

This application is a continuation of non-provisional patent applicationSer. No. 16/130,775, filed Sep. 13, 2018, which claims the benefit ofprovisional patent application No. 62/662,410, filed Apr. 25, 2018,which are hereby incorporated by reference herein in their entireties.

BACKGROUND

This relates generally to head-mounted devices, and, more particularly,to head-mounted devices with displays and image sensors.

Electronic devices often include displays and image sensors.Particularly when high-resolution images are being displayed for aviewer, it may be burdensome to display images at full resolution acrossan entire display. Foveation techniques involve displaying only criticalportions of an image at full resolution and can help reduce the burdenson a display system. In some cases, images of the environment of theuser may be displayed on the display. However, it may be burdensome touse the image sensor to obtain high-resolution images of the user'sentire environment.

SUMMARY

An electronic device such as a head-mounted device may have a display.In some cases, the display may be a transparent display so that a usermay observe real-world objects through the display whilecomputer-generated content is overlaid on top of the real-world objectsby presenting computer-generated images on the display. The display mayalso be an opaque display that blocks light from real-world objects whena user operates the head-mounted device. In this type of arrangement, apass-through camera may be used to display real-world objects to theuser.

The pass-through camera may capture images of the real world and thereal-world images may be displayed on the display for viewing by theuser. Additional computer-generated content (e.g., text, game-content,other visual content, etc.) may optionally be overlaid over thereal-world images to provide an augmented reality environment for theuser.

The display may be a foveated display. Using a gaze-tracking system inthe head-mounted device, the device may determine which portion of thedisplay is being viewed directly by a user. A user will be lesssensitive to artifacts and low resolution in portions of the displaythat lie within the user's peripheral vision than portions of thedisplay that are being directly viewed. Accordingly, the device maydisplay different portions of an image with different resolutions.

The pass-through camera may capture some high-resolution image data fordisplaying on the display. However, only low-resolution image data maybe needed to display low-resolution images in the periphery of theuser's field of view on the display. Therefore, the pass-through cameramay only capture high-resolution images that correspond to the portionof the user's field-of-view that is being directly viewed and maycapture lower resolution image data that corresponds to the real-worldobjects in the user's peripheral vision. Adjusting the pass-throughcamera to only capture high-resolution image data in selected portionsof the user's field of view may reduce processing burden and powerconsumption within the head-mounted device.

There are a number of possible arrangements for the pass-through camerathat allow the camera module to selectively capture high-resolutionimages. For example, the front-facing camera may include an image sensorwith two or more pixel densities, a distortion lens, and/or one or moreplanar or curved mirrors. Any of the components in the camera module maybe adjusted to change which portion of a scene is captured withhigh-resolution image data.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram of an illustrative head-mounted device inaccordance with an embodiment.

FIG. 2 is a top view of an illustrative head-mounted device inaccordance with an embodiment.

FIG. 3 is a diagram showing how high-resolution images may be displayedin a first portion of a user's field of view whereas low-resolutionimages may be displayed in a second portion of a user's field of view inaccordance with an embodiment.

FIG. 4 is a cross-sectional side view of an illustrative camera modulethat includes an image sensor with a varying pixel density that ispositioned by positioning equipment in accordance with an embodiment.

FIG. 5 is a top view of an illustrative image sensor of the typeincluded in FIG. 4 in accordance with an embodiment.

FIG. 6 is a cross-sectional side view of an illustrative camera modulethat includes a distortion lens that is positioned by positioningequipment in accordance with an embodiment.

FIG. 7 is a cross-sectional side view of an illustrative camera modulethat includes a curved mirror that is positioned by positioningequipment in accordance with an embodiment.

FIG. 8 is a cross-sectional side view of an illustrative camera modulethat includes an image sensor with a varying pixel density and adeformable mirror in accordance with an embodiment.

FIG. 9 is a cross-sectional side view of an illustrative camera modulethat includes an image sensor with a fixed pixel density and adeformable mirror in accordance with an embodiment.

FIG. 10 is a cross-sectional side view of an illustrative camera modulethat includes an image sensor with a varying pixel density and a planarmirror that is positioned by positioning equipment in accordance with anembodiment.

FIG. 11 is a cross-sectional side view of an illustrative camera modulethat includes an image sensor with a fixed pixel density and a planarmirror that is positioned by positioning equipment in accordance with anembodiment.

FIG. 12 is a cross-sectional side view of an illustrative camera modulethat includes a lens that is positioned by positioning equipment inaccordance with an embodiment.

FIG. 13 is a cross-sectional side view of an illustrative camera modulethat includes an image sensor with a varying pixel density and a lensthat is positioned by positioning equipment in accordance with anembodiment.

FIG. 14 is a cross-sectional side view of an illustrative camera modulethat includes a curved mirror and an image sensor with a varying pixeldensity that is positioned by positioning equipment in accordance withan embodiment.

FIG. 15 is a cross-sectional side view of an illustrative camera modulethat includes an image sensor and a lens in a housing that is positionedby positioning equipment in accordance with an embodiment.

FIG. 16 is a cross-sectional side view of an illustrative camera modulethat includes a first image sensor for capturing high-resolution images,a second image sensor for capturing low-resolution images, and abeam-splitter in accordance with an embodiment.

FIG. 17 is a cross-sectional side view of an illustrative camera modulethat includes an image sensor and a lens that may change shape tocontrol how light is directed to the image sensor in accordance with anembodiment.

FIG. 18 is a flow chart of illustrative operations involved in operatinga head-mounted device with a gaze-tracking system and a front-facingcamera in accordance with an embodiment.

DETAILED DESCRIPTION

Head-mounted devices and other devices may be used for virtual realityand augmented reality systems. These devices may include portableconsumer electronics (e.g., portable electronic devices such as cellulartelephones, tablet computers, glasses, other wearable equipment),head-up displays in cockpits and vehicles, display-based equipment(e.g., projectors, televisions), etc. Devices such as these may includetransparent displays and other optical components. Device configurationsin which virtual reality and/or augmented reality content is provided toa user with a head-mounted display are described herein as an example.This is, however, merely illustrative. Any suitable equipment may beused in providing a user with virtual reality and/or augmented realitycontent.

A head-mounted device that is worn on the head of a user may be used toprovide a user with computer-generated content that is overlaid on topof real-world content. With some head-mounted devices, the real-worldcontent may be viewed directly by a user (e.g., by observing real-worldobjects through a transparent display panel or through an opticalcoupler in a transparent display system that merges light fromreal-world objects with light from a display panel). Other head-mounteddevices may use configurations in which images of real-world objects arecaptured by a forward-facing camera and displayed for a user on adisplay. A forward-facing camera that captures images of the real-worldand displays the images on the display may be referred to as apass-through camera.

The pass-through camera may be capable of capturing high-resolutionimages to display to the user. However, a user will be less sensitive toartifacts and low resolution in portions of the display that lie withinthe user's peripheral vision than portions of the display that are beingdirectly viewed. Therefore, to reduce the processing burden and powerconsumption involved in operating the pass-through camera, thepass-through camera may only capture high-resolution images thatcorrespond to where the user is directly looking. Other portions of thecaptured image (that correspond to the user's peripheral vision) mayhave a lower resolution.

A schematic diagram of an illustrative head-mounted device is shown inFIG. 1. As shown in FIG. 1, head-mounted device 10 (sometimes referredto as head-mounted display 10) may have control circuitry 50. Controlcircuitry 50 may include storage and processing circuitry forcontrolling the operation of head-mounted device 10. Circuitry 50 mayinclude storage such as hard disk drive storage, nonvolatile memory(e.g., electrically-programmable-read-only memory configured to form asolid state drive), volatile memory (e.g., static or dynamicrandom-access-memory), etc. Processing circuitry in control circuitry 50may be based on one or more microprocessors, microcontrollers, digitalsignal processors, baseband processors, power management units, audiochips, graphics processing units, application specific integratedcircuits, and other integrated circuits. Software code may be stored onstorage in circuitry 50 and run on processing circuitry in circuitry 50to implement control operations for head-mounted device 10 (e.g., datagathering operations, operations involving the adjustment of componentsusing control signals, etc.).

Head-mounted device 10 may include input-output circuitry 52.Input-output circuitry 52 may be used to allow data to be received byhead-mounted device 10 from external equipment (e.g., a tetheredcomputer, a portable device such as a handheld device or laptopcomputer, or other electrical equipment) and to allow a user to providehead-mounted device 10 with user input. Input-output circuitry 52 mayalso be used to gather information on the environment in whichhead-mounted device 10 is operating. Output components in circuitry 52may allow head-mounted device 10 to provide a user with output and maybe used to communicate with external electrical equipment.

As shown in FIG. 1, input-output circuitry 52 may include a display suchas display 26. Display 26 may be used to display images for a user ofhead-mounted device 10. Display 26 may be a transparent display so thata user may observe real-world objects through the display whilecomputer-generated content is overlaid on top of the real-world objectsby presenting computer-generated images on the display. A transparentdisplay may be formed from a transparent pixel array (e.g., atransparent organic light-emitting diode display panel) or may be formedby a display device that provides images to a user through a beamsplitter, holographic coupler, or other optical coupler (e.g., a displaydevice such as a liquid crystal on silicon display). Alternatively,display 26 may be an opaque display that blocks light from real-worldobjects when a user operates head-mounted device 10. In this type ofarrangement, a pass-through camera may be used to display real-worldobjects to the user. The pass-through camera may capture images of thereal world and the real-world images may be displayed on the display forviewing by the user. Additional computer-generated content (e.g., text,game-content, other visual content, etc.) may optionally be overlaidover the real-world images to provide an augmented reality environmentfor the user. When display 26 is opaque, the display may also optionallydisplay entirely computer-generated content (e.g., without displayingreal-world images) to provide a virtual reality environment for theuser.

The head-mounted device may optionally include adjustable componentsstacked in series with display 26. For example, the head-mounted devicemay include an adjustable polarizer (e.g., a polarizer with switchesthat allow selected regions of the adjustable polarizer to be configuredto serve as vertical-pass linear polarizers, horizontal-pass linearpolarizers, or non-polarizing regions), tunable lenses (e.g., liquidcrystal tunable lenses, tunable lenses based on electrooptic materials,tunable liquid lenses, microelectromechanical systems tunable lenses, orother tunable lenses), an adjustable color filter (e.g., anadjustable-color-cast light filter that can be adjusted to exhibitdifferent color casts and/or a monochromatic adjustable-intensity lightfilter that has a single color cast), and/or an adjustable opacitysystem (e.g., a layer with an adjustable opacity for providing adarkened background if the display is transparent). There may be anysuitable number of display pixels in display 26 (e.g., 0-1000,10-10,000, 1000-1,000,000, 1,000,000 to 10,000,000, more than 1,000,000,fewer than 1,000,000, fewer than 10,000, fewer than 100, etc.).

Input-output circuitry 52 may include components such as input-outputdevices 60 for gathering data and user input and for supplying a userwith output. Devices 60 may include a gaze-tracker such as gaze-tracker62 (sometimes referred to as a gaze-tracking system or a gaze-trackingcamera) and a camera such as camera 64.

Gaze-tracker 62 may include a camera and/or other gaze-tracking systemcomponents (e.g., light sources that emit beams of light so thatreflections of the beams from a user's eyes may be detected) to monitorthe user's eyes. Gaze-tracker(s) 62 may face a user's eyes and may tracka user's gaze. A camera in the gaze-tracking system may determine thelocation of a user's eyes (e.g., the centers of the user's pupils), maydetermine the direction in which the user's eyes are oriented (thedirection of the user's gaze), may determine the user's pupil size(e.g., so that light modulation and/or other optical parameters and/orthe amount of gradualness with which one or more of these parameters isspatially adjusted and/or the area in which one or more of these opticalparameters is adjusted based on the pupil size), may be used inmonitoring the current focus of the lenses in the user's eyes (e.g.,whether the user is focusing in the near field or far field, which maybe used to assess whether a user is day dreaming or is thinkingstrategically or tactically), and/or other gaze information. Cameras inthe gaze-tracking system may sometimes be referred to as inward-facingcameras, gaze-detection cameras, eye-tracking cameras, gaze-trackingcameras, or eye-monitoring cameras. If desired, other types of imagesensors (e.g., infrared and/or visible light-emitting diodes and lightdetectors, etc.) may also be used in monitoring a user's gaze. The useof a gaze-detection camera in gaze-tracker 62 is merely illustrative.

Cameras such as front-facing camera(s) 64 (sometimes referred to asfront-facing camera module 64 or camera module 64) may be used tocapture images of the real-world environment surrounding the user. Forexample, one or more front-facing cameras 64 may be used to captureimages of real-world objects in front of a user and on the left andright sides of a user's field of view. The images of real-world objectsthat are gathered in this way may be presented for the user on display26 and/or may be processed by control circuitry 50 to determine thelocations of electronic devices (e.g., displays, etc.), people,buildings, and other real-world objects relative to the user. Thereal-world environment may also be analyzed using image processingalgorithms. Information from camera 64 may be used in controllingdisplay 26.

Front-facing camera 64 may serve as a pass-through camera that obtainsimages of the real-world environment of the user. The real-world imagescorresponding to the user's field of view (as determined by thegaze-tracker and the position of the head-mounted device) are thendisplayed on display 26. In this way, the user perceives that they areviewing the real world (by replicating real-world viewing with thepass-through camera and display).

In addition to adjusting components such as display 26 based oninformation from gaze-tracker 62 and/or front-facing cameras 64, controlcircuitry 50 may gather sensor data and user input from otherinput-output circuitry 52 to use in controlling head-mounted device 10.As shown in FIG. 1, input-output devices 60 may include position andmotion sensors 66 (e.g., compasses, gyroscopes, accelerometers, and/orother devices for monitoring the location, orientation, and movement ofhead-mounted device 10, satellite navigation system circuitry such asGlobal Positioning System circuitry for monitoring user location, etc.).Using sensors 66, for example, control circuitry 50 can monitor thecurrent direction in which a user's head is oriented relative to thesurrounding environment. Movements of the user's head (e.g., motion tothe left and/or right to track on-screen objects and/or to viewadditional real-world objects) may also be monitored using sensors 66.

Input-output devices 60 may also include other sensors and input-outputcomponents 70 (e.g., ambient light sensors, force sensors, temperaturesensors, touch sensors, buttons, capacitive proximity sensors,light-based proximity sensors, other proximity sensors, strain gauges,gas sensors, pressure sensors, moisture sensors, magnetic sensors,microphones, speakers, audio components, haptic output devices,light-emitting diodes, other light sources, etc.). Circuitry 52 mayinclude wired and wireless communications circuitry 74 that allowshead-mounted device 10 (e.g., control circuitry 50) to communicate withexternal equipment (e.g., remote controls, joysticks and other inputcontrollers, portable electronic devices, computers, displays, etc.) andthat allows signals to be conveyed between components (circuitry) atdifferent locations in head-mounted device 10. Head-mounted device 10may include any other desired components. For example, the head-mounteddevice may include a battery.

The components of head-mounted device 10 may be supported by ahead-mountable support structure such as illustrative support structure16 of FIG. 2. Support structure 16 may have the shape of a frame of apair of glasses (e.g., left and right temples and other frame members),may have a helmet shape, or may have another head-mountableconfiguration. When worn on the head of a user, the user may viewreal-world objects such as object 30 through display 26 inconfigurations where display 26 is a transparent display. Inconfigurations where display 26 is opaque, the user's eyes 12 may beblocked from viewing object 30. Display 26 is supported by supportstructure 16 and is placed in front of user eyes 12 when worn on thehead of the user.

Support structure 16 may support additional components at additionallocations such as locations 38, 40, and 42. For example, components maybe mounted on the front of support structure 16 in location 38.Front-facing cameras 64 and/or sensors and other components ininput-output circuitry 52 may be mounted in location 38. The componentsin location 38 may be used to detect the positions of real-world objects(e.g., object 30) and/or for capturing images of the real-world. Object30 may include natural and manmade objects, people, buildings, sourcesof glare such as reflective objects, the sun, lights, etc.

Input-output devices 60 such as position and motion sensors 66, lightdetectors, or other desired input-output devices may be mounted inlocation 40. Components in location 40 may face the environment of theuser (e.g., outward facing components facing away from the user). Incontrast, components in location 42 may face the user (e.g., inwardfacing components facing the user). Input-output devices 60 such asgaze-tracker 62 (image sensors), speakers (e.g., ear speakers) or otheraudio components that play audio (e.g., audio associated withcomputer-generated images and/or other content that is being displayedusing display 26, etc.) or other desired input-output devices may bemounted in location 42.

Display 26 may be a foveated display. Using gaze-tracking (e.g., usinggaze-tracker 62 to capture information on the location of a user's gazeon display 26), device 10 can determine which portion of display 26 isbeing viewed only by a user's peripheral vision and which portion ofdisplay 26 is being viewed directly (non-peripherally) by a user (e.g.,in the centermost 5° of the user's field of view corresponding to thefovea of the user's eyes where visual acuity is elevated). A user willbe less sensitive to artifacts and low resolution in portions of display26 that lie within the user's peripheral vision than portions of display26 that are being directly viewed. Accordingly, device 10 may displaydifferent portions of an image with different resolutions.

FIG. 3 shows a field of view 90 corresponding to the field of view ofthe user while wearing head-mounted device 10. The user may look atregion 94 of the display 26. Accordingly, images on the display inregion 94 may be presented with a relatively high resolution. Ifdesired, images may be presented on the display at a high resolutionacross the user's entire field of view. However, to conserve processingburden and power consumption, regions of the display that the user isnot directly viewing (e.g., the user's peripheral vision) such as region92 may present low-resolution images (e.g., at a lower resolution thanin region 94).

In some cases (e.g., when the device is in a pass-through mode), display26 displays real-world images corresponding to what the user would seein the absence of the head-mounted device. When the device is in thepass-through mode, the entire display may display real-world images thatare captured by a camera in the device (e.g., front-facing camera 64 inFIG. 1). In this mode, the display may present high-resolution imagescorresponding to the real world in region 94. Therefore, front-facingcamera 64 must be capable of capturing high-resolution images. However,only low-resolution image data is needed to display the low-resolutionimages in region 92.

If desired, front-facing camera 64 may capture only high-resolutionimages. Control circuitry 50 may then process the image data to presentthe high-resolution images in region 94 while presenting lowerresolution images in region 92. In other words, some of the capturedhigh-resolution image data is discarded to present lower resolutionimages in region 92. However, capturing excess image data (that willultimately be discarded) may use valuable processing and powerresources. So, instead of capturing excess high-resolution image data,front-facing camera 64 may instead only capture high-resolution imagesthat correspond to the portion of the user's field-of-view that is beingdirectly viewed. Front-facing camera 64 captures lower resolution imagedata that corresponds to the real-world objects in the user's peripheralvision. Adjusting front-facing camera 64 to only capture high-resolutionimage data in selected portions of the user's field of view may reduceprocessing burden and power consumption within head-mounted device 10.

There are a number of possible arrangements for camera module 64(sometimes referred to as an outward-facing camera or an imaging system)that allow the camera module to selectively capture high-resolutionimages. For example, the front-facing camera may include an image sensorwith two or more pixel densities, a distortion lens, and/or one or moreplanar or curved mirrors. Any of the components in the camera module maybe adjusted to change which portion of a scene is captured withhigh-resolution image data.

FIG. 4 is a cross-sectional side view of an illustrative camera module64 with an image sensor that has a non-constant pixel density across thesensor. As shown in FIG. 4, camera module 64 includes an image sensor102 having a first pixel density portion 103A and a second pixel densityportion 103B. The first and second pixel density portions 103A and 103Bhave different respective pixel densities. In particular, the secondpixel density portion 103B of image sensor 102 may have a greater pixeldensity than first pixel density portion 103A. Second pixel densityportion 103B may therefore be referred to as high pixel density portion103B and first pixel density portion 103A may be referred to as lowpixel density portion 103A. High pixel density portion 103B may havemore pixels-per-inch (PPI) than low pixel density portion 103A. The highpixel density portion 103B will capture image data having a higherresolution than the low pixel density portion 103A.

Camera module 64 may include one or more lenses such as lens 104 forfocusing incident light corresponding to the captured real-world scene(e.g., light 80) onto image sensor 102. Some of the incident light(e.g., a first portion of the captured scene) will be received by highpixel density portion 103B of the image sensor whereas some of theincident light (e.g., a second portion of the captured scene) will bereceived by low pixel density portion 103A of the image sensor.High-resolution image data will therefore be obtained of the firstportion of the captured scene, whereas low-resolution image data will beobtained of the second portion of the captured scene.

Camera module 64 may also include positioning equipment 106 foradjusting the position of image sensor 102. In particular, positioningequipment 106 may adjust the position of image sensor 102 to adjustwhich portion of the incoming light (e.g., which portion of the capturedscene) is imaged by the high pixel density portion of the image sensor.Arrows 108 show how the image sensor may be shifted laterally (e.g.,within the XY-plane) by positioning equipment 106. Positioning equipment106 may position image sensor 102 underneath lens 104 based on sensorinformation (e.g., information from gaze-tracker 62 and/or position andmotion sensors 66). This sensor information may be used to determine apoint of gaze of the user (e.g., the point to which the user islooking). Positioning equipment 106 may then move image sensor 102 suchthat high pixel density portion 103B of the image sensor receives lightcorresponding to the point of gaze of the user (e.g., the portion of thescene at which the user is looking).

Positioning equipment 106 may include any desired components. Forexample, the positioning equipment may include one or more of a motor(e.g., a servomotor, a geared motor, a brushless motor, etc.), a linearelectromagnetic actuator (e.g., a solenoid), a piezoelectric device, anelectroactive polymer, a pneumatic actuator, and any other suitable typeof actuator. Positioning equipment 106 may be configured to move imagesensor 102 within the XY-plane, move image sensor 102 vertically alongthe Z-axis, and/or tilt image sensor 102 (such that the image sensor isat an angle relative the XY-plane).

If desired, the components of camera module 64 may be formed in housing100 (sometimes referred to as camera module housing 100). Housing 100may support image sensor 102, lens 104, and/or positioning equipment106.

Image sensor 102 may have an increased pixel area to account for themovement of the image sensor underneath lens 104. In particular, it isdesirable for image sensor 102 to capture all of the incoming lightcorresponding to the captured scene, regardless of the position of highpixel density portion 103B. When, high pixel density pixel portion 103Bis centered underneath lens 104 (as in FIG. 4), the periphery of theimage sensor (102P) may not receive incident light. However, consider anexample where the image sensor is shifted laterally along the X-axis(e.g., to align high pixel density portion 103B under the right-mostedge of lens 104) in FIG. 4. The peripheral portion of the image sensormay be shifted to now receive incident light (e.g., from the left-mostedge of lens 104). Therefore, ensuring that image sensor 102 has alarger area than is necessary to capture all of the incident light(while the sensor is centered) ensures that all of the incident lightwill still be captured even if the sensor is shifted to move the highpixel density portion of the sensor to the edges of the captured scene.

FIG. 5 is a top view showing show image sensor 102 may have a high pixeldensity region 103B and a low pixel density region 103A. As shown inFIG. 5, the low pixel density region 103A may laterally surround thehigh pixel density region 103B. This example is merely illustrative. Ifdesired, image sensor 102 may include any desired number of differentpixel density regions, with each pixel density region having any desiredshape and any desired pixel density. There may be a gradual transitionbetween the pixel densities of adjacent pixel density regions ifdesired.

The example in FIGS. 4 and 5 of image sensor 102 having different pixeldensity regions is merely illustrative. If desired, a camera module mayinstead use a distortion lens to magnify a portion of a captured scene,thus obtaining high-resolution image data for the portion of thecaptured scene. An arrangement of this type is shown in FIG. 6.

As shown in FIG. 6, camera module 64 includes an image sensor 102 with afixed pixel density across the image sensor. Similar to FIG. 4, imagesensor 102 may receive incoming light 80 from a lens. However, in FIG. 4light is provided to image sensor 102 with a uniform angular resolution.In contrast, in FIG. 6 light is provided to image sensor 102 withvarying angular resolution. In particular, light in the center of thelens (for example) may be spread across a greater corresponding area ofimage sensor 102 than light at the periphery of the lens. As shown inFIG. 6, light corresponding to a first area 110 of the lens may bespread onto a larger area 112 of the image sensor. In other words, theportion of the captured scene received at area 110 of lens 104D ismagnified by lens 104D. The light received at area 110 is spread overmore pixels than if the light was not distorted by the lens. Having morepixels to capture image data for the same area of the incoming lightmeans that the image data will be of a higher resolution than image datafor the other portions of the image sensor.

To summarize, lens 104D may distort incoming light to optically stretch(e.g., magnify) a selected portion of the captured scene over a largerpixel area than if the light was not distorted (e.g., lens 104Dselectively increases angular resolution of a selected portion thecaptured scene). The image sensor therefore obtains high-resolutionimage data for the selected portion of the captured scene. The remainingportions of the captured scene are not optically stretched (and may beoptically compressed). The image sensor therefore obtains low-resolutionimage data (with at least a lower resolution than the high-resolutionimage data) for the remaining portions of the captured scene.

Camera module 64 may also include positioning equipment 106 foradjusting the position of lens 104D. In particular, positioningequipment 106 may adjust the position of lens 104D to adjust whichportion of the incoming light (e.g., which portion of the capturedscene) is optically stretched by the lens for obtaining high-resolutionimage data. Arrows 108 show how the lens may be shifted laterally (e.g.,within the XY-plane) by positioning equipment 106. Positioning equipment106 may be configured to move distortion lens 104D within the XY-plane,move distortion lens 104D vertically along the Z-axis, and/or tiltdistortion lens 104D (such that the distortion lens is at an anglerelative the XY-plane). Positioning equipment 106 may positiondistortion lens 104D based on sensor information (e.g., information fromgaze-tracker 62 and/or position and motion sensors 66). This sensorinformation may be used to determine a point of gaze of the user (e.g.,the point to which the user is looking). Positioning equipment 106 maythen move distortion lens 104D such that optically stretched portion ofthe captured image (e.g., area 110) corresponds to the point of gaze ofthe user (e.g., the portion of the scene at which the user is looking).

In yet another embodiment, an additional optical component may beincluded in camera module 64 to enable image sensor 102 to generatehigh-resolution image data. As shown in FIG. 7, a mirror such as mirror114 may be interposed in the optical path between lens 104 and imagesensor 102. Mirror 114 may have any desired shape (e.g., curved orplanar). Additionally, more than one mirror (e.g., an array of mirrors)may be included in the optical path between lens 104 and image sensor102 if desired.

In FIG. 7, image sensor 102 may be an image sensor with a fixed pixeldensity (similar to as shown in FIG. 6) and lens 104 may not be adistortion lens (e.g., similar to lens 104 in FIG. 4). However, mirror114 may distort incident image light (similar to the distortion lens104D of FIG. 6). In other words, mirror 114 may distort the incidentlight from lens 104 to optically stretch (e.g., magnify) a selectedportion of the captured scene over a larger pixel area than if the lightwas not distorted. The image sensor therefore obtains high-resolutionimage data for the selected portion of the captured scene. The remainingportions of the captured scene are not optically stretched (and may beoptically compressed). The image sensor therefore obtains low-resolutionimage data (with at least a lower resolution than the high-resolutionimage data) for the remaining portions of the captured scene.

Camera module 64 may also include positioning equipment 106 foradjusting the position of mirror 114. In particular, positioningequipment 106 may adjust the position of mirror 114 to adjust whichportion of the incoming light (e.g., which portion of the capturedscene) is optically stretched by the mirror for obtaininghigh-resolution image data. Arrows 116 show how the mirror may berotated (e.g., rotated about a central axis 118) by positioningequipment 106. Positioning equipment 106 may also be configured to movemirror 114 within the XY-plane, move mirror 114 vertically along theZ-axis, and/or tilt mirror 114. Positioning equipment 106 may positionmirror 114 based on sensor information (e.g., information fromgaze-tracker 62 and/or position and motion sensors 66). This sensorinformation may be used to determine a point of gaze of the user (e.g.,the point to which the user is looking). Positioning equipment 106 maythen move mirror 114 such that the optically stretched portion of thecaptured image corresponds to the point of gaze of the user (e.g., theportion of the scene at which the user is looking).

In yet another embodiment, shown in FIG. 8, a deformable mirror such asdeformable mirror 120 may be interposed in the optical path between lens104 and image sensor 102. In FIG. 8, image sensor 102 has two or morepixel density regions such as high pixel density region 103B and lowpixel density region 103A. Deformable mirror 120 may determine whichportion of the captured scene is directed to the high pixel densityregion 103B. In particular, deformable mirror 120 may have two or morestates in which incoming light 80 from lens 104 is directed to differentlocations on image sensor 102. As shown in FIG. 8, deformable mirror 120has a first state in which the mirror has a first shape 120A and asecond state in which the mirror has a second shape 120B. Positioningequipment 106 may adjust deformable mirror 120 between different shapes(such as 120A and 120B) to control which portion of the captured sceneis directed to high pixel density region 103B of the image sensor.

Positioning equipment 106 may control the shape of deformable mirror 120based on sensor information (e.g., information from gaze-tracker 62and/or position and motion sensors 66). This sensor information may beused to determine a point of gaze of the user (e.g., the point to whichthe user is looking). Positioning equipment 106 may then control theshape of deformable mirror 120 such that the portion of the capturedimage corresponding to the point of gaze of the user (e.g., the portionof the scene at which the user is looking) is directed to the high pixeldensity region of the image sensor.

The use of a single mirror in FIGS. 7 and 8 is merely illustrative. Inboth FIGS. 7 and 8, an array of mirrors may be used to redirect lightbetween lens 104 and image sensor 102. Each mirror in the array ofmirrors may be independently controlled (e.g., rotated as in FIG. 7 ordeformed as in FIG. 8) by positioning equipment 106.

The aforementioned examples are merely illustrative, and variousmodifications may be made to the camera modules. In particular, anydesired combinations of high distortion lenses, lenses without highdistortion (sometimes referred to as low distortion lenses), deformablemirrors, rotatable mirrors, image sensors with constant pixel density,and image sensors with variable pixel density may be used in the cameramodule. Additionally, positioning equipment may move any of thecomponents in the camera module in any desired manner.

FIG. 9 shows a camera module with a lens 104 and deformable mirror 120that is controlled by positioning equipment 106 (similar to the cameramodule in FIG. 8). However, in FIG. 8 image sensor 102 has a varyingpixel density, whereas in FIG. 9 image sensor 102 has a fixed pixeldensity. In FIG. 9, either lens 104 or mirror 120 may optically stretchincoming light to create the high-resolution image data. For example,lens 104 may be a high distortion lens (as in FIG. 6) that magnifies aportion of the captured scene. Alternatively, mirror 120 may distort aselected portion of the captured scene (similar to as discussed inconnection with FIG. 7). Positioning equipment 106 may control the shapeof deformable mirror 120 and/or may control the position of lens 104.Positioning equipment 106 may control the components in camera module 64based on sensor information (e.g., information from gaze-tracker 62and/or position and motion sensors 66).

In yet another embodiment, a planar mirror such as planar mirror 114 maybe interposed in the optical path between lens 104 and image sensor 102,as shown in FIG. 10. In this embodiment, lens 104 may be a lowdistortion lens and planar mirror 114 may not distort the incominglight. Therefore, image sensor 102 may be a variable pixel density imagesensor with high pixel density portion 103B and low pixel densityportion 103A to enable capture of high-resolution image data. The planarmirror 114 is positioned to direct a desired portion of the capturedscene to high pixel density portion 103B. The remaining portions of thecaptured scene are directed to the low pixel density portion 103A. Theimage sensor therefore obtains high-resolution image data for thedesired portion of the captured scene and low-resolution image data(with at least a lower resolution than the high-resolution image data)for the remaining portions of the captured scene.

Camera module 64 may also include positioning equipment 106 foradjusting the position of planar mirror 114. In particular, positioningequipment 106 may adjust the position of planar mirror 114 to adjustwhich portion of the incoming light (e.g., which portion of the capturedscene) is received by high pixel density region 103B. Arrows 116 showhow the mirror may be rotated (e.g., rotated about a central axis 118)by positioning equipment 106. Positioning equipment 106 may also beconfigured to move mirror 114 within the XY-plane, move mirror 114vertically along the Z-axis, and/or tilt mirror 114. Positioningequipment 106 may position mirror 114 based on sensor information (e.g.,information from gaze-tracker 62 and/or position and motion sensors 66).This sensor information may be used to determine a point of gaze of theuser (e.g., the point to which the user is looking). Positioningequipment 106 may then move mirror 114 such that the portion of thecaptured image directed to high pixel density region 103B corresponds tothe point of gaze of the user (e.g., the portion of the scene at whichthe user is looking).

FIG. 11 shows an embodiment similar to the embodiment of FIG. 10, withboth embodiments having a rotatable planar mirror 114. However, in FIG.10 lens 104 is low distortion lens, whereas in FIG. 11, a distortionlens 104D magnifies a selected portion of the image. As shown in FIG.11, distortion lens 104D optically stretches a portion of the capturedscene (similar to as discussed in connection with FIG. 6). Positioningequipment may control the position of planar mirror 114 and/or theposition of distortion lens 104D based on sensor information (e.g.,information from gaze-tracker 62 and/or position and motion sensors 66).FIG. 11 shows an image sensor with a fixed pixel density, but the imagesensor may have a varying pixel density if desired.

FIG. 12 show an embodiment similar to the embodiment of FIG. 7, with alens 104, a mirror 114 that magnifies a portion of the incoming light,and a fixed pixel density image sensor 102. Lens 104 may provide lightwith a uniform angular resolution to curved mirror 114. Mirror 114 thenmagnifies a portion of the light and redirects the light towards imagesensor 102. However, in FIG. 7 positioning equipment controlled theposition of mirror 114 to control which portion of the scene wasmagnified for high-resolution image data. In FIG. 12, in contrast,positioning equipment 106 controls the position of lens 104 to controlwhich portion of the scene is directed to the magnifying portion ofmirror 114.

Arrows 108 show how the lens may be shifted laterally (e.g., within theXY-plane) by positioning equipment 106. Positioning equipment 106 may beconfigured to move lens 104 within the XY-plane, move lens 104vertically along the Z-axis, and/or tilt lens 104 (such that the lens isat an angle relative the XY-plane). Positioning equipment 106 mayposition lens 104 based on sensor information (e.g., information fromgaze-tracker 62 and/or position and motion sensors 66). This sensorinformation may be used to determine a point of gaze of the user (e.g.,the point to which the user is looking). Positioning equipment 106 maythen move lens 104 such that the portion of the captured image directedto the magnifying portion of mirror 114 corresponds to the point of gazeof the user (e.g., the portion of the scene at which the user islooking).

FIG. 13 shows an embodiment similar to the embodiment of FIG. 4, with alens 104 and a variable pixel density image sensor 102 having a highpixel density region 103B and a low pixel density region 103A. Lens 104may provide light with a uniform angular resolution to variable pixeldensity image sensor 102. However, in FIG. 4 positioning equipmentcontrolled the position of image sensor 102 to control which portion ofthe scene was received by high pixel density region 103B. In FIG. 13, incontrast, positioning equipment 106 controls the position of lens 104 tocontrol which portion of the scene is directed to high pixel densityregion 103B of the image sensor.

Arrows 108 show how the lens may be shifted laterally (e.g., within theXY-plane) by positioning equipment 106. Positioning equipment 106 may beconfigured to move lens 104 within the XY-plane, move lens 104vertically along the Z-axis, and/or tilt lens 104 (such that the lens isat an angle relative the XY-plane). Positioning equipment 106 mayposition lens 104 based on sensor information (e.g., information fromgaze-tracker 62 and/or position and motion sensors 66). This sensorinformation may be used to determine a point of gaze of the user (e.g.,the point to which the user is looking). Positioning equipment 106 maythen move lens 104 such that the portion of the captured image thatcorresponds to the point of gaze of the user (e.g., the portion of thescene at which the user is looking) is directed to the high pixeldensity region 103B of the image sensor.

In yet another embodiment, shown in FIG. 14, camera module 64 mayinclude lens 104, mirror 114, and variable pixel density image sensor102, similar to the embodiment of FIG. 10. In FIG. 10 mirror 114 isplanar, whereas in FIG. 14, mirror 114 is curved. Lens 104 in FIG. 14may be a low distortion lens. Image sensor 102 may be a variable pixeldensity image sensor with high pixel density portion 103B and low pixeldensity portion 103A to enable capture of high-resolution image data.The mirror 114 is positioned to direct the captured scene to the imagesensor. A first portion of the captured scene is received and imaged byhigh pixel density portion 103B, and the remaining portions of thecaptured scene are received and imaged by low pixel density portion103A. The image sensor therefore obtains high-resolution image data fora portion of the captured scene and low-resolution image data (with atleast a lower resolution than the high-resolution image data) for theremaining portions of the captured scene. The curved mirror 114 mayoptionally magnify a portion of the image for an additional increase ofresolution of the image data.

Camera module 64 may also include positioning equipment 106 foradjusting the position of image sensor 102. In particular, positioningequipment 106 may adjust the position of image sensor 102 to adjustwhich portion of the incoming light (e.g., which portion of the capturedscene) is imaged by the high pixel density portion of the image sensor.Arrows 108 show how the image sensor may be shifted laterally (e.g.,within the YZ-plane) by positioning equipment 106. Positioning equipment106 may position image sensor 102 based on sensor information (e.g.,information from gaze-tracker 62 and/or position and motion sensors 66).This sensor information may be used to determine a point of gaze of theuser (e.g., the point to which the user is looking). Positioningequipment 106 may then move image sensor 102 such that high pixeldensity portion 103B of the image sensor receives light corresponding tothe point of gaze of the user (e.g., the portion of the scene at whichthe user is looking).

FIG. 15 shows yet another embodiment for camera module 64. In FIG. 15,camera module 64 includes a high distortion lens 104D that focuses lighton image sensor 102. Housing 100 supports image sensor 102 and lens104D. The image sensor is a fixed pixel density image sensor. Distortionlens 104D magnifies a portion of the captured scene so thathigh-resolution image data is obtained for the portion of the capturedscene. In FIG. 6, positioning equipment 106 moved lens 104D to controlwhich portion of the captured scene was magnified. In contrast,positioning equipment 106 in FIG. 15 changes the position of housing 100(as shown by arrows 124 for example) to control which portion of thecaptured scene is magnified. Positioning equipment 106 can rotate orshift the position of housing 100 to change the direction lens 104D andimage sensor 102 point. Positioning equipment 106 may position housing100 based on sensor information (e.g., information from gaze-tracker 62and/or position and motion sensors 66). This sensor information may beused to determine a point of gaze of the user (e.g., the point to whichthe user is looking). Positioning equipment 106 may then move housing100 such that lens 104D magnifies incident light corresponding to thepoint of gaze of the user (e.g., the portion of the scene at which theuser is looking).

The example in FIG. 15 of the movable housing being used with astationary high distortion lens and a stationary fixed pixel densityimage sensor is merely illustrative. In general, any of the previousembodiments may include positioning equipment that can rotate or shiftthe position of housing 100. For example, a movable housing as shown inFIG. 15 may include a variable pixel density image sensor, a planarmirror, a curved mirror, a deformable mirror, and/or a low distortionlens, and the positions of any of these components may be adjusted bypositioning equipment.

In yet another embodiment, shown in FIG. 16, camera module 64 includes abeam splitter such as beam splitter 126. Beam splitter 126 (e.g., aprism) splits incoming light 80 onto two image sensors: image sensor102H and image sensor 102L. Image sensor 102H may have a higherresolution (e.g., more pixels per inch) than image sensor 102L. Imagesensor 102H may therefore sometimes be referred to as a high-resolutionimage sensor and image sensor 102L may sometimes be referred to as alow-resolution image sensor. Control circuitry in the head-mounteddevice (e.g., control circuitry 50 in FIG. 1) may dynamically selectwhich portions of high-resolution image sensor 102H and/orlow-resolution image sensor 102L to read out. The image data may then becombined to form a single image with high-resolution image data indesired portions and low-resolution image data in the remainingportions. Control circuitry 50 may select which portions of each sensorto read out based on sensor information (e.g., information fromgaze-tracker 62 and/or position and motion sensors 66) such as point ofgaze information.

FIG. 17 shows another embodiment for camera module 64. In FIG. 17, thecamera module includes a deformable lens. As shown in FIG. 17, imagesensor 102 and lens 128 are formed in housing 100 of camera module 64.Lens 128 is a deformable lens (sometimes referred to as a shape-changinglens or adaptable lens). Lens 128 may be controlled (e.g., by equipment106) to take a desired shape. For example, lens 128 may change between afirst shape 129A and a second shape 129B. The different shapes of lens128 may each have a different profile of angular resolution whilemaintaining the same focus. This way, the captured scene may be focusedonto image sensor 102 (regardless of the shape of the lens). However,the different shapes of the deformable lens may allow different portionsof the captured scene to be magnified (for high-resolution image data).For example, when the lens has shape 129A a portion 130A of the lens maymagnify a first portion of the incoming light (e.g., increase theangular resolution of the light relative to surrounding lens portions).When the lens has shape 129B a portion 130B of the lens (that isdifferent than portion 130A) may magnify a second, different portion ofthe incoming light. In this way, the shape of the lens may be controlledto select a portion of the incoming light to optically stretch.Therefore, the shape of the lens may be controlled to obtainhigh-resolution image data of a selected portion of the captured scene.The lens may be changed between any desired number of shapes (e.g., two,three, four, more than four, more than ten, less than twenty, etc.),with each shape having an associated resolution of image data obtainedby image sensor 102.

Lens 128 may be formed in any desired manner that allows the lens tochange shape. For example, the lens may be a liquid lens that changesshape based on liquid volume. The lens may be a liquid crystal lens thatchanges shape based on a voltage. The lens may includemicroelectromechanical systems (MEMS) if desired.

FIG. 18 is a flowchart of illustrative method steps that may beperformed during operations of a head-mounted device such ashead-mounted device 10 in FIG. 1. As shown in FIG. 18, at step 202control circuitry (e.g., control circuitry 50 in FIG. 1) may gatherinformation from input devices in the head-mounted device. Controlcircuitry 50 may gather information from any desired input devices. Forexample, control circuitry 50 may gather information from gaze-trackingcamera 62, position and motion sensors 66, or any other desired inputdevices. The information gathered at step 202 may include point of gazeinformation (e.g., information indicating where the user is looking).

Next, at step 204, control circuitry 50 may adjust front-facing camera64 based on the information obtained during step 202 (e.g., the point ofgaze information). The control circuitry may adjust the front-facingcamera in any desired manner (e.g., by adjusting the position of a lens,the shape of a lens, the position of a mirror, the shape of a mirror,the position of an image sensor, or the position of a camera modulehousing). The control circuitry may adjust the front-facing camera suchthat the front-facing camera obtains high-resolution image data for aportion of the scene that corresponds to the point of gaze of the userand low-resolution image data for portions of the scene that correspondto the periphery of the user's field of view. After the front-facingcamera is adjusted, the front-facing camera may capture image data thatis then displayed on display 26 of the head-mounted device.

The foregoing is merely illustrative and various modifications can bemade to the described embodiments. The foregoing embodiments may beimplemented individually or in any combination.

What is claimed is:
 1. A head-mounted device, comprising: a head-mountedsupport structure; a pass-through camera that is supported by thehead-mounted support structure and that is configured to capture animage of a real-world scene, wherein the image has a first-resolutionportion and a second-resolution portion that has a higher resolutionthan the first-resolution portion; a display that is supported by thehead-mounted support structure and that is configured to display theimage; a gaze-tracking system that is configured to obtain point of gazeinformation; and control circuitry that is configured to adjust a lensin the pass-through camera based on the point of gaze information,wherein the lens outputs light with varying angular resolution andwherein adjusting the lens adjusts which portion of the real-world scenecorresponds to the second-resolution portion of the image.
 2. Thehead-mounted device defined in claim 1, wherein the pass-through cameracomprises an image sensor that is configured to capture the image. 3.The head-mounted device defined in claim 2, wherein the lens isconfigured spread light incident in a first portion of the lens across agreater corresponding area of the image sensor than light incident in asecond portion of the lens.
 4. The head-mounted device defined in claim2, wherein the lens is configured spread light incident in a center ofthe lens across a greater corresponding area of the image sensor thanlight incident in a periphery of the lens.
 5. The head-mounted devicedefined in claim 1, wherein the lens increases an angular resolution ofa portion of incident light from the real-world scene that correspondsto the second-resolution portion of the image.
 6. A head-mounted device,comprising: a head-mounted support structure; a display that issupported by the head-mounted support structure and that is configuredto display an image; a gaze-tracking system that is configured to obtainpoint of gaze information; and a camera module comprising: an imagesensor that is configured to capture the image based on incident lightfrom real-world objects; and a lens that magnifies some but not all ofthe incident light onto the image sensor, wherein the lens is configuredto be adjusted based on the point of gaze information.
 7. Thehead-mounted device defined in claim 6, wherein light in a first area ofthe lens is spread onto a second area of the image sensor and whereinthe second area is greater than the first area.
 8. The head-mounteddevice defined in claim 6, wherein the image has a first-resolutionportion and a second-resolution portion having a higher resolution thanthe first-resolution portion and wherein the incident light magnified bythe lens corresponds to the second-resolution portion of the image.
 9. Ahead-mounted device, comprising: a head-mounted support structure; anoutward-facing camera that is supported by the head-mounted supportstructure and that is configured to use an image sensor to capture animage of a real-world scene, wherein the image has a first-resolutionportion and a second-resolution portion that has a higher resolutionthan the first-resolution portion; a gaze-tracking system that issupported by the head-mounted support structure and that includes aninward-facing camera, wherein the gaze-tracking system is configured toobtain point of gaze information based on images captured by theinward-facing camera; a display that is supported by the head-mountedsupport structure and that is configured to display the image of thereal-world scene; and control circuitry that is configured to adjustwhich portion of the real-world scene corresponds to thesecond-resolution portion of the image based on the point of gazeinformation.
 10. The head-mounted device defined in claim 9, wherein thecontrol circuitry is configured to physically move a component in theoutward-facing camera to adjust which portion of the real-world scenecorresponds to the second-resolution portion of the image.
 11. Thehead-mounted device defined in claim 10, wherein the component comprisesa component selected from the group consisting of: an image sensor, amirror, a lens, and a camera module housing.
 12. The head-mounted devicedefined in claim 10, wherein the component is a distortion lens thatmagnifies incident light from a first portion of the real-world scenethat corresponds to the first-resolution portion of the image less thanincident light from a second portion of the real-world scene thatcorresponds to the second-resolution portion of the image.